fix(internal): XML (TwiML) serialization drops characters outside XML 1.0 in all SDK generators by cadesark · Pull Request #18423 · fern-api/fern

cadesark · 2026-10-21T16:49:15Z

Description

Linear ticket: N/A (Twilio TwiML parity tracker, issue 31)

Twilio reported that a value containing a control character (e.g. \x01) is emitted verbatim into TwiML by the generated PHP SDK, producing XML that no parser accepts (the legacy SDK had the same bug). Every XML serializer in the seven SDK generators now drops characters that are not legal in XML 1.0 before escaping, so the same input yields the same, valid output in every language.

Kept: #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]. Dropped: everything else (\x00–\x08, \x0B, \x0C, \x0E–\x1F, \uFFFE, \uFFFF, lone surrogates). Applies to text nodes, attribute values and (where the serializer supports them) comments.

Changes Made

TypeScript core/xml/serialize.ts: escapeXml strips INVALID_XML_CHARS (/[^\u0009\u000A\u000D\u0020-\uD7FF\uE000-\uFFFD\u{10000}-\u{10FFFF}]/gu) before escaping; text/attribute escaping behaviour otherwise unchanged (quotes stay literal in text).
Python xml_utilities.py: escape/quoteattr wrap xml.sax.saxutils and remove _INVALID_XML_CHARS first.
Java XmlWriter.escape: iterates code points, skips !isXmlChar(cp), appends the rest with appendCodePoint (so surrogate pairs survive, lone surrogates are dropped).
Go core/xml.go: strings.Map filter with isXmlChar(r rune) before xml.EscapeText.
C# XmlUtils.Serialize: serializes a filtered clone (WithoutInvalidXmlChars) covering XText, XComment and attributes; StripInvalidXmlChars uses XmlConvert.IsXmlChar + XmlConvert.IsXmlSurrogatePair and only allocates a StringBuilder when something has to be removed (net462/net8.0/netstandard2.0 compatible).
PHP XmlUtils::stripInvalidXmlChars (preg_replace with the u flag), applied to text, comments and attributes (setDomAttribute).
Ruby-v2 Internal::Xml::Utils: INVALID_XML_CHARS removed in escape_text; escape_attribute builds on it.
Regression tests in each language ("a\x00b\x1Fc\td" → abc\td, attribute "man\x01" → man), unreleased changelog entries for all 7 generators, regenerated *-xml-twiml seed fixtures.
Updated README.md generator (not applicable)

Testing

Unit tests added/updated
Manual testing completed
TS: core-utilities vitest + regenerated ts-xml-twiml (3/3 fixtures) and its xml unit tests (24 passed)
Python: pytest tests/utils/test_xml_utilities.py (35 passed), mypy/ruff
Java: generator-utils compiles, regenerated java-xml-twiml gradle tests pass; XmlElement.builder("Say").attribute("voice","man\u0001").text("a\u0000b\u001Fc\td") → <Say voice="man">abc d</Say>
Go: go test ./internal/generator/sdk/core/...
C#: regenerated csharp-xml-twiml (both fixtures), dotnet test 255 passed
PHP: regenerated php-xml-twiml, phpunit 150 tests OK, phpstan no errors
Ruby: regenerated ruby-xml-twiml, 701 runs / 0 failures, rubocop clean

Link to Devin session: https://app.devin.ai/sessions/311fe13ae5434245ada665a9b824809b
Open in Devin Desktop: https://app.devin.ai/desktop/session/311fe13ae5434245ada665a9b824809b?variant=devin
Requested by: @cadesark

… 1.0 in all SDK generators Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ng; Ruby test and C# System.Text import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-21T16:49:19Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

nitpickybot

AI Review Summary

Adds XML 1.0 Char filtering across seven SDK generators plus regenerated seed fixtures. The approach is consistent and well-tested, but the C# StripInvalidXmlChars has an inverted surrogate-pair check that silently drops every valid astral character (emoji, CJK ext), and the Go filter drops the UTF-8 replacement rune that strings.Map emits for invalid bytes rather than the original byte.

🔴 1 critical issue(s)
🟡 2 warning(s)
🔵 1 suggestion(s)

Review Scope

Mode: Full pull request
Range: Full pull request through c30bf4544eda489d955ac897b16480c9fd5541ac
Current range coverage: Complete
Cumulative pull request coverage: Complete

This review is complete for the current scope. Addressing all critical and warning findings above satisfies the review; suggestions are optional. Request another review only if subsequent changes introduce new behavior or materially expand the pull request's scope.

devin-ai-integration

✅ Devin resolved all 3 findings on `c30bf45`

Fixed by Devin (3)

Invalid comments still break XML output
Root sibling comments remain invalid
Namespace declarations bypass character filtering

View all findings in Devin Review

github-actions · 2026-10-21T17:16:08Z

SDK Generation Benchmark Results

Comparing PR branch against median of 5 nightly run(s) on main (latest: 2026-10-21T04:06:11Z).

Full benchmark table (click to expand)

Generator	Spec	main (generator)	main (E2E)	PR (generator)	Delta
csharp-sdk	square	98s (n=5)	119s (n=5)	68s	-30s (-30.6%)
go-sdk	square	119s (n=5)	245s (n=5)	135s	+16s (+13.4%)
java-sdk	square	204s (n=5)	267s (n=5)	241s	+37s (+18.1%)
php-sdk	square	71s (n=5)	N/A	61s	-10s (-14.1%)
python-sdk	square	129s (n=5)	179s (n=5)	123s	-6s (-4.7%)
ruby-sdk-v2	square	89s (n=5)	131s (n=5)	91s	+2s (+2.2%)
ts-sdk	square	137s (n=5)	143s (n=5)	139s	+2s (+1.5%)

main (generator): generator-only time via --skip-scripts (includes Docker image build, container startup, IR parsing, and code generation — this is the same Docker-based flow customers use via fern generate). main (E2E): full customer-observable time including build/test scripts (nightly baseline, informational). Delta is computed against generator-only baseline.
⚠️ = generation exited with a non-zero exit code (timing may not reflect a successful run).
Baseline from nightly runs on main (latest: 2026-10-21T04:06:11Z). Trigger benchmark-baseline to refresh.
Last updated: 2026-10-21 21:22 UTC

- C#: explicit high/low surrogate guards, sibling comments sanitized - TS/Python/Java/Ruby/Go: comments filtered through the shared helper - PHP: namespace declaration URIs filtered before setAttributeNS - Java: rename c -> codePoint; regression tests in all generators Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…strip-invalid-chars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # generators/typescript/utils/core-utilities/src/core/xml/serialize.ts # seed/ts-sdk/ts-xml-twiml/forward-compatible-enums/src/core/xml/serialize.ts # seed/ts-sdk/ts-xml-twiml/no-custom-config/src/core/xml/serialize.ts # seed/ts-sdk/ts-xml-twiml/serde-layer/src/core/xml/serialize.ts

…strip-invalid-chars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> # Conflicts: # generators/csharp/base/src/asIs/XmlUtils.Template.cs # seed/csharp-sdk/csharp-xml-twiml/no-custom-config/src/SeedApi/Core/XmlUtils.cs # seed/csharp-sdk/csharp-xml-twiml/use-undiscriminated-unions/src/SeedApi/Core/XmlUtils.cs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cadesark and others added 2 commits October 21, 2026 16:31

fix(internal): XML (TwiML) serialization drops characters outside XML…

ef4b6f2

… 1.0 in all SDK generators Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

fix(internal): regenerate XML (TwiML) seeds with invalid-char strippi…

c30bf45

…ng; Ruby test and C# System.Text import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cadesark requested a review from iamnamananand996 as a code owner October 21, 2026 16:49

devin-ai-integration Bot assigned cadesark Oct 21, 2026

nitpickybot Bot reviewed Oct 21, 2026

View reviewed changes

devin-ai-integration Bot reviewed Oct 21, 2026

View reviewed changes

cadesark and others added 3 commits October 21, 2026 17:28

style(python): ruff-format xml_utilities invalid-char pattern

0c16fb3

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

ci: re-run after cli-v2 check.test.ts 5s timeout flake

badb06e

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

iamnamananand996 approved these changes Oct 21, 2026

View reviewed changes

cadesark and others added 3 commits October 21, 2026 20:18

ci: retrigger (php-sdk-tests clone auth flake)

4f2fbfa

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cadesark merged commit 516cdeb into main Oct 21, 2026
177 of 178 checks passed

cadesark deleted the devin/1792599947-xml-strip-invalid-chars branch October 21, 2026 21:22

Navigation Menu

Repository navigation

fix(internal): XML (TwiML) serialization drops characters outside XML 1.0 in all SDK generators - #18423

fix(internal): XML (TwiML) serialization drops characters outside XML 1.0 in all SDK generators#18423
cadesark merged 8 commits into
mainfrom
devin/1792599947-xml-strip-invalid-chars

cadesark commented Oct 21, 2026 •

edited by devin-ai-integration Bot

Loading

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026

Uh oh!

nitpickybot Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cadesark commented Oct 21, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes Made

Testing

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026

Uh oh!

nitpickybot Bot left a comment

Choose a reason for hiding this comment

AI Review Summary

Review Scope

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

✅ Devin resolved all 3 findings on c30bf45

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SDK Generation Benchmark Results

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cadesark commented Oct 21, 2026 •

edited by devin-ai-integration Bot

Loading

devin-ai-integration Bot left a comment •

edited

Loading

✅ Devin resolved all 3 findings on `c30bf45`

github-actions Bot commented Oct 21, 2026 •

edited

Loading